High fidelity remote controller device for digital living room

ABSTRACT

Described herein is an intelligent remote controlling device (e.g. a mobile phone). The device can include a six-axis motion sensor to accurately track three dimensional hand motions. For example, the sensors can include a three-axis accelerometer and a three-axis gyroscope. The remote control device can also include a processing unit integrated with the motion sensors in a single module. The processing unit can convert data regarding the hand motion to data regarding a cursor motion for a cursor that will be displayed on a screen of an electronic device. The processing unit can be integrated with the motion sensors in a single module (e.g. an integrated circuit chip (IC)). The processing unit can include at least two modes of functionality corresponding to different types of hand motion: a one to one mode where the cursor directly tracks the hand motion and a non-linear mode that filters data from the motion sensors to eliminate hand jitter.

CROSS REFERENCE TO RELATED APPLICATIONS

The subject application is a continuation of U.S. patent applicationSer. No. 13/909,940 filed Jun. 4, 2013 entitled “HIGH FIDELITY REMOTECONTROLLER DEVICE FOR DIGITAL LIVING ROOM” which is a continuation ofU.S. patent application Ser. No. 13/164,747 filed Jun. 20, 2011 entitled“HIGH FIDELITY REMOTE CONTROLLER DEVICE FOR DIGITAL LIVING ROOM” whichclaims priority to U.S. Provisional Patent Application Ser. No.61/439,822 filed Feb. 4, 2011 entitled “NOVEL REMOTE CONTROLLERS WITHMOTION SENSORS FOR CONTROLLING AND NAVIGATING TELEVISION SETS AND 3DCOMPUTER USER INTERFACES, AND NOVEL PICO PROJECTORS AND USERAUTHENTICATION DEVICES WITH MOTION SENSORS” the entireties of each areincorporated by reference herein.

TECHNICAL FIELD

Described herein is a remote controller device that can employ motionsensors to facilitate mode switching functions to accommodate differenttypes of hand motions.

BACKGROUND

A remote controller device can be utilized to detect hand motion andcorrespond to the detected hand motion to motion of a cursor on ascreen. When a user tries to point the remote controller device at thescreen, for example to select a small icon or link, the remotecontroller device can detect the intended pointing motion, but can alsodetect unintended hand jitter. In the case of pointing, the hand jittercan be of the same magnitude of the intended hand motion, and the remotecontroller device can have trouble distinguishing the intended motionfrom the hand jitter.

SUMMARY

The following presents a simplified summary of the claimed subjectmatter in order to provide a basic understanding of some aspectsdescribed herein. This summary is not an extensive overview, and is notintended to identify key/critical elements or to delineate the scope ofthe claimed subject matter. Its sole purpose is to present some conceptsin a simplified form as a prelude to the more detailed description thatis presented later.

Described herein are systems, methods and apparatuses for a remotecontrolling device. The remote controlling device can include at leasttwo motion sensors that can detect a hand motion. For example, thesensors can include a three-axis accelerometer and a three-axisgyroscope to accurately trace three dimensional motions. The remotecontrol device can also include a processing unit that can convert dataregarding the hand motion to data regarding a cursor motion. Theprocessing unit can be integrated with the motion sensors in a singlemodule. The processing unit can include at least two modes offunctionality corresponding to different types of hand motion: a one toone mode where the cursor directly tracks the hand motion and anon-linear mode that filters data from the motion sensors to eliminatehand jitter. The remote controlling device can also include a radiofrequency (RF) that can transmit the data regarding the cursor movementto an electronic device (e.g., a television screen).

The following description and annexed drawings set forth certainillustrative aspects of the specification. These aspects are indicative,however, of but a few of the various ways in which the principles of thespecification can be employed. Other advantages and novel features ofthe specification will become apparent from the following detaileddescription of the specification when considered in conjunction with thedrawings.

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the subject disclosureare described with reference to the following figures, wherein likereference numerals refer to like parts throughout the various viewsunless otherwise specified.

FIG. 1 is a schematic system block diagram of an embodiment of a remotecontrol device.

FIG. 2 is a schematic system block diagram of an embodiment of a remotecontrol device.

FIG. 3 is a schematic system block diagram of an embodiment of a remotecontrol device.

FIG. 4 is a schematic system block diagram of an embodiment of a remotecontrol device.

FIG. 5 is a schematic system block diagram of an embodiment of a remotecontrol device.

FIG. 6 is a schematic system block diagram of an embodiment of a remotecontrol device.

FIG. 7 is a schematic system block diagram of an embodiment of a remotecontrol device.

FIG. 8 is a process flow diagram of a method for motion processing inlinear mode.

FIG. 9 is a process flow diagram of a method for motion processing inpointing mode.

FIG. 10 is a process flow diagram of a method for intelligentlytransitioning between motion processing modes in a remote controllerdevice.

FIG. 11 is a schematic system block diagram of button assignments for astate machine.

FIG. 12 is a schematic system block diagram of an embodiment of a statemachine.

FIG. 13 is a schematic system block diagram of an embodiment of a statemachine.

FIG. 14 is a schematic system block diagram of an embodiment of a statemachine.

FIG. 15 is an exemplary state diagram utilized by an embodiment of astate machine.

FIGS. 16-19 are exemplary state transition diagrams as part of the statediagram illustrated in FIG. 15.

FIG. 20 is an exemplary illustration of a signal processing data flow inan embodiment of an intelligent remote control algorithm.

DETAILED DESCRIPTION

Various non-limiting embodiments of a remote control device and methodsutilized with the remote control device are described herein. In thefollowing description, numerous specific details are set forth in orderto provide a thorough understanding of one or more embodiments. Oneskilled in the relevant art will recognize, however, that the techniquesdescribed herein can be practiced without one or more of the specificdetails, or with other methods, components, materials, etc., and is notlimited by these specific details and examples. In other instances,well-known structures, materials, and/or operations are not shown ordescribed in detail to avoid obscuring certain aspects.

Reference throughout this specification to “one embodiment,” or “anembodiment,” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment. Thus, the appearances of the phrase “in oneembodiment,” or “in an embodiment,” in various places throughout thisspecification are not necessarily all referring to the same embodiment.Furthermore, the particular features, structures, or characteristics maybe combined in any suitable manner in one or more embodiments.

The word “exemplary” is used herein to mean serving as an example,instance, or illustration. For the avoidance of doubt, the subjectmatter disclosed herein is not limited by such examples. Any aspectdescribed herein as “exemplary” is not necessarily to be construed aspreferred or advantageous over other, nor is it meant to precludeequivalent exemplary structures and techniques known to those ofordinary skill in the art. Furthermore, to the extent that the terms“includes,” “has,” “contains,” and other similar words are used ineither the detailed description or the claims, such terms are intendedto be inclusive—in a manner similar to the term “comprising” as an opentransition word—without precluding any additional or other elements.

As used in this application, the term “or” is intended to mean aninclusive “or” rather than an exclusive “or.” Therefore, unlessspecified otherwise, or clear from context, “X employs A or B” isintended to mean any of the natural inclusive permutations. That is, ifX employs A; X employs B; or X employs both A and B, then “X employs Aor B” is satisfied under any of the foregoing instances. In addition,the articles “a” and “an” as used in this application and the appendedclaims should generally be construed to mean “one or more” unlessspecified otherwise or clear from context to be directed to a singularform.

Referring now to FIG. 1, illustrated is a schematic system block diagramof a remote control device 100. The remote control device 100 can, forexample, be utilized in connection with a television, computer, or thelike. The remote control device 100 can be utilized to control anyelectronic device with a display. The remote control device 100 can haveany number of buttons. For example, the remote control device 100 can befree of buttons or have one, two, three, etc. buttons. In oneembodiment, the remote control device 100 can have a single button. Thesingle button can be utilized to both turn an associated device on and,once the associated device is on, to indicate gestures. For example, thebutton can be pressed and the remote control can sense that the user isintending to make a gesture.

Remote control device 100 can sense hand motions and convert the handmotions to signals understood by an electronic device. For example, thehand motions can translate to cursor movements on a screen of anelectronic device. To sense the hand motions, the remote control devicecan include motion sensors 102. Although two motion sensors areillustrated here, it will be understood that any number of motionsensors can be utilized to detect hand motion. According to anembodiment, the motion sensors 102 can include a gyroscope. According toanother embodiment, the motion sensors 102 can include an accelerometer.According to a further embodiment, the motion sensors 102 can include agyroscope and an accelerometer. According to a further embodiment, themotion sensors 102 can include a compass. According to anotherembodiment, the motion sensors 102 can include six axes. In one aspect,the motion sensors 102 can be embodied on a single integrated circuit(IC) chip.

The motion sensors 102 can detect hand motion, including, for example,gestures made while holding the remote control device 100. Examples ofgestures the motion sensors 102 can detect include, but are not limitedto, a tap, a shake, a hand written letter, character and/or symbol, etc.According to an embodiment, the motion sensors 102 can detect a gesture,and the gesture can be utilized in switching motion capture modes forthe remote control device 100.

Remote control device 100 can have multiple modes of motionfunctionality. According to an embodiment, the remote control device 100can operate in a linear mode. The linear mode can be a one to one motiontracking mode in which the hand motion is accurately captured. Thelinear mode can be useful, for example, for Asian character input,drawing, handwriting recognition, and the like.

According to another embodiment, the remote control device 100 canoperate in a pointing mode. The pointing mode can be a non-linear mode.In the pointing mode, any hand motion, including undesirable jitter canbe detected. For example, jitter can be due to hand jitter, buttonpushing, and the like. In the pointing mode, the hand motion can befiltered to reduce effects caused by jitter. The pointing mode can beuseful, for example, for selecting links on Web pages, clicking icons ina document, or the like.

The modes can be implemented by a processing unit 104, which can becoupled to the motion sensors 102. The processing unit 104 can receivedata from the motion sensors 102 indicating a hand motion was detected.Upon receiving the data, the processing unit 104 can convert the datafrom the motion sensor into data corresponding to the display of anelectronic device. For example, the processing unit 104 can processthree-dimensional data from the motion sensors 102 regarding the handmotion into two dimensional data able to be displayed on an electronicdevice display. For example, the processing unit 104 can convert thedata from the motion sensors 102 indicating a hand motion into dataindicating an on-screen cursor movement, for example, for a televisionscreen or computer monitor.

According to an embodiment, the motion sensors 102 can be embodied in asingle module and the processing unit 104 can be embodied in anothermodule. According to another embodiment, both the motion sensors 102 andthe processing unit 104 can be embodied in a single module. In a furtherembodiment, the motion sensors 102 can be embodied in a single moduleand the processing unit 104 and a radio frequency (RF) unit (describedbelow) can be embodied on another module. According to anotherembodiment, the processing unit 104 can be split across two modules,with a first part of the processing unit embodied on a single modulewith the motion sensors 102 and the second part of the processing unitembodied on another module with the RF unit (described below). In afurther embodiment, the motion sensors 102, the processing unit 104 andthe RF unit (described below) can be embodied on a single module.

Referring now to FIG. 2, illustrated is a schematic system block diagramof a remote control device 200. The remote control device 200 can, forexample, be utilized in connection with a television, computer, or thelike. The remote control device 200 can be utilized to control anyelectronic device with a display. The remote control device 200 can haveany number of buttons. For example, the remote control device 200 can befree of buttons or have one, two, three, etc. buttons.

Remote control device 200 can include motion sensors 102 coupled to aprocessing unit 104 to detect hand motions and convert data related tothe hand motions to data useable by a electronic device, for example atelevision or a computer. The remote control device 200 includes amotion processing unit (MPU) 202 that can integrate the motion sensors102 with the processing unit 104. According to an embodiment, the MPU202 can be situated on a single integrated circuit (IC), so that themotion sensors 102 and the processing unit 104 are embodied on a singleIC chip.

Although two motion sensors 102 are illustrated as part of the MPU, itwill be understood that any number of motion sensors can be utilized todetect hand motion. According to an embodiment, the motion sensors 102can include a gyroscope. According to another embodiment, the motionsensors 102 can include an accelerometer. According to an embodiment,the motion sensors 102 can include an accelerometer. According toanother embodiment, the motion sensors 102 can include a compass.According to a further embodiment, the motion sensors 102 can include agyroscope and an accelerometer. According to another embodiment, themotion sensors 102 can include six axes.

The motion sensors 102 can detect hand motion, including, for example,gestures made while holding the remote control device 100. Examples ofgestures the motion sensors 102 can detect include, but are not limitedto, a tap, a shake, a hand written letter, character and/or symbol, etc.According to an embodiment, the motion sensors 102 can detect a gesture,and the gesture can be utilized in switching motion capture modes forthe remote control device 200.

Remote control device 200 can have multiple modes of motionfunctionality. According to an embodiment, the remote control device 200can operate in a linear mode. The linear mode can be a one to one motiontracking mode in which the hand motion is accurately captured. Thelinear mode can be useful, for example, for Asian character input,drawing, handwriting recognition, and the like.

According to another embodiment, the remote control device 200 canoperate in a pointing mode. The pointing mode can be a non-linear mode.In the pointing mode, any hand motion, including undesirable jitter canbe detected. For example, jitter can be due to hand jitter, buttonpushing, and the like. In the pointing mode, the hand motion can befiltered to reduce effects caused by jitter. The pointing mode can beuseful, for example, for selecting links on Web pages, clicking icons ina document, or the like.

A processing unit 104 can implement the mode for the remote controldevice 200. The motion sensors 102 can be coupled to the processing unit104 so that the processing unit 104 can receive data from the motionsensors 102 indicating a hand motion was detected. Upon receiving thedata, the processing unit 104 can convert the data from the motionsensor into data corresponding to the display of an electronic device.For example, the processing unit 104 can process three-dimensional datafrom the motion sensors 102 regarding the hand motion into twodimensional data able to be displayed on an electronic device display.For example, the processing unit 104 can convert the data from themotion sensors 102 indicating a hand motion into data indicating anon-screen cursor movement, for example, for a television screen orcomputer monitor.

Referring now to FIG. 3, illustrated is a schematic system block diagramof a remote control device 300. The remote control device 300 can, forexample, be utilized in connection with a television, computer, or thelike. The remote control device 300 can be utilized to control anyelectronic device with a display. The remote control device 300 can haveany number of buttons. For example, the remote control device 300 can befree of buttons or have one, two, three, etc. buttons.

Remote control device 300 can include a motion processing unit (MPU) 202that can integrate motion sensors, including one or more accelerometers302 and one or more gyroscopes 304, and a processing unit 104. Themotion sensors can also include a compass (not illustrated). The MPU 202can be a single module that can detect hand motions and process the handmotions into data usable by an electronic device (e.g., a cursor on atelevision screen). According to an embodiment, the MPU 202 can besituated on a single integrated circuit (IC), so that the accelerometer302, gyroscope 304 and the processing unit 104 are embodied on a singleIC chip.

According to an embodiment, the accelerometer 302 and the gyroscope 304can have six axes. For example, accelerometer 302 can be a three axisaccelerometer and gyroscope 304 can be a three axis gyroscope. Utilizingthe six axes, the MPU 202 can calculate a three dimensional (3D)orientation corresponding to a hand motion with a high accuracy.

For example, a person can make a hand motion with the remote control.The hand motion can be a tap, a shake, a hand written letter, character,and/or a symbol, or the like. The accelerometer 302 and gyroscope 304can detect the hand motion. For example, accelerometer 302 can detectacceleration of the remote control device and the gyroscope 304 candetect orientation of the remote control device. A three axisaccelerometer 302 can detect acceleration in 3D, while a three axisgyroscope can detect orientation in 3D. The processing unit 104 can take3D data from the accelerometer 302 and the gyroscope 304 correspondingto the hand motion and convert the 3D data to 2D data, for example,corresponding to a cursor movement on a screen (e.g., a televisionscreen or a computer monitor).

According to another embodiment, the hand motion can be utilized toswitch motion capture modes in the remote control device 300. Remotecontrol device 300 can have multiple modes of motion functionality.According to an embodiment, the remote control device 300 can operate ina linear mode. The linear mode can be a one to one motion tracking modein which the hand motion is accurately captured. The linear mode can beuseful, for example, for Asian character input, drawing, handwritingrecognition, and the like.

According to another embodiment, the remote control device 300 canoperate in a pointing mode. The pointing mode can be a non-linear mode.In the pointing mode, any hand motion, including undesirable jitter canbe detected. For example, jitter can be due to hand jitter, buttonpushing, and the like. In the pointing mode, the hand motion can befiltered to reduce effects caused by jitter. The pointing mode can beuseful, for example, for selecting links on Web pages, clicking icons ina document, or the like.

Referring now to FIG. 4, illustrated is a schematic system block diagramof a remote control device 400. The remote control device 400 can, forexample, be utilized in connection with a television, computer, or thelike. The remote control device 400 can be utilized to control anyelectronic device with a display. The remote control device 400 can haveany number of buttons. For example, the remote control device 400 can befree of buttons or have one, two, three, etc. buttons.

Remote control device 400 can include a motion control unit 202. Forexample, the motion control unit 202 can include motion sensors and aprocessing unit. The motion sensors can include an accelerometer, agyroscope, and/or a compass. According to an embodiment, theaccelerometer can be a three axis accelerometer and the gyroscope can bea three axis gyroscope. The motion control unit 202 can be a moduleincorporating motion sensors 102 and a processing unit 104.

zThe MPU can detect a hand motion through the motion sensors. Data fromthe motion sensors can be sent to the processing unit, and can betransformed into data usable by an electronic device, like a televisionscreen, a computer monitor, or the like, by the processing unit. Theprocessing unit can, for example, transform three dimensional (3D) datafrom the motion sensors indicating a hand motion into two dimensional(2D) data for use by the electronic device, such as a cursor motion on ascreen.

The remote control device 400 can also include a radio frequency (RF)unit 402 coupled to the MPU 202. The RF unit 402 can receive the 2D datafor use by the electronic device, such as the cursor motion on thescreen from the MPU 202. The RF unit 402 can transmit the 2D data foruse by the electronic device via a wireless protocol. Although notshown, the electronic device can be a television, a computer, or anyelectronic device with a screen or other user interface. The wirelessprotocol can include WiFi™, Bluetooth™, Radio Frequency for ConsumerElectronics (RF4CE), and the like.

According to an embodiment, the MPU and the RF unit can be situated onthe same integrated circuit (IC) chip. According to another embodiment,the MPU and the RF unit can be situated on different IC chips.

Remote control units 100-400 as illustrated in FIG. 1-4 can operateunder multiple modes of motion functionality. For example, processingunit 104 can operate differently under the multiple modes. In atraditional computing environment, there are two different requirementsfor a remote control device: detection of large motions and detection ofsmall motions. Remote control devices 100-400 can employ a linear modefor detection of large motions and a pointing mode for detection ofsmall motions.

The linear mode can be utilized for detection of large motions. Thelinear mode can be a one to one motion tracking mode. The linear modecan be utilized, for example, for Asian characters, drawing,handwriting, or the like. In the linear mode, the overall motion isgreater than any extraneous motion, such as hand jitter. For example,when the hand motion is a hand written letter, the overall motion of theentire letter is greater than any unintended hand jitter. Therefore, themotion can still be tracked without being affected by extraneous motionslike hand jitter. The linear motion can still accurately reconstruct thehand motion.

On the other hand, hand jitter and other extraneous motions can affectmotions in the pointing mode. The pointing mode can detect any handmotion, including the intended hand motion and accompanying jitter. Inthe pointing mode, the hand motion is a small motion, such as selectinglinks on a Web page, clicking icons in a document, and the like. Thehand jitter can be of approximately the same order as the hand motion,so, unlike the linear mode, the pointing mode cannot match the handmotion one to one to the intended action. Accordingly, the pointing modeis a non linear mode. In the pointing mode, hand motions can be filteredto reduce the extraneous effects, for example, caused by unwantedmotions or disturbances, such as hand jitter, button pushing, or thelike.

For example, in the pointing mode, a user can point the remote controldevice 100-400 to select small icons or to navigate between small icons,for example with regard to a Web browser or a menu tree system. Sincesome users have a significant hand jitter, the remote control device100-400 can interpret user intention and help to track the target iconin spite of the significant hand jitter.

According to an aspect, a user of the remote control device 100-400 isable to switch between modes (e.g., the linear mode and the pointingmode) at will. The remote control device can switch between modes, forexample, by detecting a user pressing a button, by detecting a certaingesture, such as tap, a shake, a handwriting symbol, or a character, orby utilizing content to decide upon the mode without user intervention.

Referring now to FIG. 5 illustrated is a schematic system block diagramof a remote control device 500 that can include a button 502 forindicating a mode switch. The remote control device 500 can switchbetween modes by detecting a user pressing a button 502. For example, auser can activate button 502 to indicate to the MPU 202 to switch modes.Although a “button” is described here, it should be understood that anyother mechanism that a user can signal is within the scope of a“button.” According to an aspect, the MPU 202 can indicate differentmodes by utilizing different cursor shapes corresponding to thedifferent modes. Based on the shape of the cursor, a user can identifythe mode of remote control device 500.

Referring now to FIG. 6, illustrated is a schematic system block diagramof a remote control device 600 employing a different mechanism forallowing a user to switch modes at will. Remote control device 600 canemploy a gesture based mode switching. For example, the sensor 602 (orthe MPU 202) can detect a motion that indicates a mode switch. Thegesture can include one or more of as tap, a shake, a handwritingsymbol, or a character. According to an embodiment, sensor 602 can beone or more motion sensors that are part of the MPU 202.

For example, the motion can be a shaking motion. The shaking motion canbe at a predefined altitude. This gesture can replace the need foractivating a button to indicate a mode switch. According to anembodiment, remote controller device 600 can be programmed with aspecific gesture indicating a mode switch. For example, the mode switchcan be triggered when a user vertically flips the remote controllerdevice 600, for example, to point up to the sky or down to the ground,then shakes the remote controller device 600 a certain number of times(e.g., three times). The MPU 202 can indicate different modes byutilizing different cursor shapes corresponding to the different modes.Based on the shape of the cursor, a user can identify the mode of remotecontrol device 600.

Referring now to FIG. 7, illustrated is a schematic system block diagramof a remote control device 700 employing a different mechanism forallowing a user to switch modes at will. Remote control device 700 canemploy a content based mode switch, illustrated as intelligencecomponent 702. The intelligence component 702 can include systemsoftware that, when executed by a processor, can make the mode switchdecision and execute the mode switch without user intervention.

For example, intelligence component 702 can detect when a cursor is ontop of an application that necessitates a specific mode. For example,the intelligence component 702 can detect that the cursor is on top of aWeb browser and determine that the Web browser requires the remotecontroller device 700 to be in the pointing mode. The intelligencecomponent 702 can detect that the Web browser includes small icons thatnecessitate the high precision pointing mode. The intelligence component702 can indicate to the MPU 202 to change the mode to the pointing modeand change the cursor to a style corresponding to the pointing mode.When the cursor is on top of a different application, for example atelevision pop up, a video pop up, or the like, the intelligencecomponent 702 can detect that the different application does not requirethe high precision pointing mode and can initiate a switch to the linearmode.

The remote controllers as described in FIGS. 1-6 can operate in multiplemodes. For example, these modes can include a linear mode and a pointingmode. The linear mode can be a one to one motion tracking mode. Thelinear mode can be useful, for example, for Asian character input,drawing, handwriting recognition, and the like. The pointing mode can bea non-linear mode that can remove undesirable hand jitter from the handmotion, for example, by filtering the hand motion to reduce effectscaused by jitter. The pointing mode can be useful, for example, forselecting links on Web pages, clicking icons in a document, or the like.

Referring now to FIG. 8, illustrated is a process flow diagram for amethod 800 of motion processing employing the linear mode. In method 800and other following methodologies, the methodologies are illustrated asschematic process flow diagrams. While, for purposes of simplicity ofexplanation, the methodologies are shown and described as a series ofacts, it is to be understood and appreciated that the order of acts isnot limiting, as some acts may occur in different orders and/orconcurrently with other acts from that shown and described herein. Forexample, those skilled in the art will understand and appreciate that amethod could alternatively be represented as a series of interrelatedstates or events, such as in a state diagram. Moreover, not allillustrated acts may be required to implement a method.

The motion processing begins at element 802, where sensors (e.g., motionsensors in a remote control device) sense a hand motion. The sensors caninclude an accelerometer, a gyroscope and/or a compass. Theaccelerometer can be a three axis accelerometer. The gyroscope can be athree axis gyroscope. The hand motions detected by the motion sensorscan include, but are not limited to a drawing, a hand written letter,character and/or symbol, etc.

Data regarding the hand motion sensed by the motion sensors can betransmitted to a processing unit, and, at element 804, a rotationalmovement is calculated from the data regarding the hand motion. Therotational movement can be transformed into cursor location coordinates(e.g. three dimensional (3D) coordinates). For example, the rotationalmovement can be transformed into 3D Euler angles.

According to an embodiment, the hand rotational movement can berepresented by the quaternion of 3D rotation. The corresponding cursorlocation coordinates can be linearly mapped to a two dimensional (2D)position. For example, the cursor location coordinates represented bythe quaternion of 3D rotation can be Yaw and Pitch Euler angles, whichcan be linearly mapped to a 2D position. The 2D position can be relatedto a position of a cursor that will be displayed on a screen of anelectronic device like a television, a computer screen, or the like. Forexample, hand rotation movement on the yaw axis can be mapped to theX-axis on a 2D screen and hand rotation movement on the pitch axis canbe mapped to the Y-axis on the 2D screen.

High fidelity one to one motion tracking can be achieved by tracking thecursor location coordinates (e.g., Euler angles), corresponding to truemotion of a user's hand, with an internal cursor image. Any differencebetween the true motion and the internal cursor angles will be utilizedto calculate a digitized delta (e.g., mouse data format). The remainingvalue can be accumulated in the internal cursor image so that there isno long term drift due to the quantization error.

In the linear mode, although hand jitter is detected with the handmotion, hand jitter is small compared to the cursor moving to indicatehand motion. Therefore, hand jitter can be ignored in the processing.

In contrast, in the pointing mode, hand jitter is not small compared tothe hand motion. Accordingly, the hand jitter can be filtered from thehand motion and removed. FIG. 9 illustrates a process flow diagram for amethod 900 of motion processing employing the pointing mode.

The method 900 begins in the same way as method 800 of FIG. 8. Themotion processing begins at element 902, where sensors (e.g., motionsensors in a remote control device) sense a hand motion. The sensors caninclude an accelerometer and a gyroscope. The accelerometer can be athree axis accelerometer. The gyroscope can be a three axis gyroscope.The hand motions detected by the motion sensors can include, but are notlimited to a tap, a shake, etc.

Data regarding the hand motion sensed by the motion sensors can betransmitted to a processing unit, and, at element 804, a rotationalmovement is calculated from the data regarding the hand motion. Therotational movement can be transformed into cursor location coordinates(e.g., 3D Euler angles).

According to an embodiment, the hand rotational movement can berepresented by the quaternion of 3D rotation. The corresponding cursorlocation coordinates can be linearly mapped to a two dimensional (2D)position. For example, the 2D position can be related to a position of acursor that will be displayed on a screen of an electronic device like atelevision, a computer screen, or the like. For example, hand rotationmovement on the yaw axis can be mapped to the X-axis on a 2D screen andhand rotation movement on the pitch axis can be mapped to the Y-axis onthe 2D screen.

High fidelity one to one motion tracking can be achieved by tracking thecursor location coordinates, corresponding to true motion of a user'shand, with an internal cursor image. Any difference between the truemotion and the internal cursor angles will be utilized to calculate adigitized delta (e.g., mouse data format). The remaining value can beaccumulated in the internal cursor image so that there is no long termdrift due to the quantization error.

The motion sensors can detect any motions from a user's hand. When theuser, for example, an elderly person or a young child, holds the remotecontrolling device the hand jitters, or makes a motion different thanthe intended motion. Accordingly, the hand jitter, or otheruncertainties, can show up in the sensed signals. If the user moves theremote controller device faster than the hand jitter, as is the case forthe linear mode, the uncertainties from hand jitter do not negativelyinfluence detection of the hand motion and/or user intention associatedwith the hand motion. However, as in the case of the pointing mode, ifthe user points at a small object (e.g., an icon or a link) with theremote controlling device, the effect of hand jitter will be apparent.

At element 906, hand jitter can be removed from the sensed hand motion.A user intention can be identified when the user points to one target,even with strong hand jitter. However, when the user intends to make afine adjustment of the pointing, with the size of the hand jitter,uncertainties with magnitudes similar to the magnitude of the fineadjustment make separating a user intended hand gesture from the handjitter.

According to an aspect, the motion sensor can detect both hand motionand jitter. Rotational movement corresponding to the hand motion can becalculated and transformed into the cursor location coordinates whilethe hand jitter is removed. Accordingly, only the hand motion will notbe shown as the cursor movement on the screen and the cursor movementwill not be affected by the hand jitter. For example, a nonlinearquadratic function can be utilized to reduce and/or eliminate the handjitter. According to an embodiment, an intelligent state machine can beemployed to remove the hand jitter.

In removing hand jitter, a basic assumption about the frequency handjitter can be made. For example, the frequency of hand jitter can beassumed to be around 5 to around 10 hertz (Hz). Accordingly, if a 200 Hzsampling rate is utilized for a motion sensor signal processing modules,a full cycle of hand jitter can take around 20 to around 40 samples.

In addition to hand jitter, button pushing can cause artifacts that canbe removed in a method similar to the removal of hand jitter. Forexample, the remote controller device can include one or more buttons.When a user pushes the buttons, some unwanted hand motion can beincluded. This unwanted hand motion can disturb the cursor on thescreen. According to an embodiment, the intelligent state machine can beemployed to remove these unwanted motions.

For example, hand jitter and/or button push motion can be detected inthe intelligence of the state machine. The true hand motion can betracked and the one to one cursor movement according to the hand motioncan be achieved.

The state machine can remove hand jitter and/or button pushing artifactsand can also intelligently detect when a user intends to transitionbetween modes. According to an embodiment, as illustrated in FIG. 10,the state machine can employ a method 1000 to intelligently transitionmodes in a remote controller device. For example, the state machine cantransition from a pointing mode to a linear mode (e.g., a moving mode ora drawing mode). The state machine can also transition from a linearmode to a pointing mode.

The method 1000 begins at element 1002 where a hand jitter window iscreated. In removing hand jitter, a basic assumption about the frequencyhand jitter can be made. For example, the frequency of hand jitter canbe assumed to be around 5 to around 10 hertz (Hz). Accordingly, if a 200Hz sampling rate is utilized for a motion sensor signal processingmodules, a full cycle of hand jitter can take around 20 to around 40samples. Based on this assumption, a hand jitter window is created inwhich the maximum and minimum of delta, a current sample minus aprevious sample, can be calculated in real time. For example, the windowcan be a 40 sample window. Logic for the window is simple to implementin software and can be executed by a hardware processor.

For example, for a remote controller device utilized with a television,the window can be a rectangular box. The two dimensional signals, e.g.yaw angle and pitch angle, can be utilized with the rectangular box. A2D max and min with a predefined margin can define the size of the handjitter window. The size of the hand jitter window can be user dependent.Additionally or alternatively, the size of the hand jitter can beadjusted in real time during the pointing mode.

During the pointing mode, the cursor is frozen even though the sensorscan detect a movement. This detected movement can be an intended gestureor merely hand jitter or another artifact. The intelligent state machinecan determine whether the detected movement is an intended gesture ormerely hand jitter or another artifact.

At element 1004, a target position is recorded for the current pointingmode. For example, this can be a position where the cursor is frozen inthe pointing mode. At element 1006, at each sample, a displacement ofthe current sensed movement from the target position is calculated. Atelement 1008, a user intention is determined. For example, if thedisplacement is consistently larger than the boundary of the window inmore than a certain number of samples (e.g., 20 samples) in onedirection, a user intention of leaving the target point is determined.At element 1010, a mode can be transitioned (e.g., from the pointingmode to the linear mode—either a moving mode or a drawing mode).

During the point mode, the cursor is frozen, even though the motionsensors can record a motion. In contrast, during the linear mode (e.g.,drawing mode or moving mode), the cursor will follow the hand motion. Inmethod 1000, the latency from the state transition from pointing mode tolinear mode is a maximum of 20 samples or 100 milliseconds.

The state machine with hysteresis can be utilized to smooth thetransition between the pointing mode and the linear mode (e.g., movingmode or drawing mode). Additionally, the state machine can also handlebutton click and/or double click intelligently.

As illustrated in FIG. 11, the state machine 1000 can be associated withbuttons having different functionalities. FIG. 11 shows an exemplaryembodiment of buttons that can be utilized for a mouse cursor displaystate machine. For example, a Selection Button 1102 can select thetarget icon. The Gesture Button 1104 can be utilized to trigger gestureactions. For example, a gesture action can include a yaw gesture tochange a Web page, a pitch gesture to scroll down a long Web page, and aroll gesture to change a system speaker volume. The Glyph Button 1106can trigger a glyph engine, which will display a trace of the glyph on ascreen and, at the same time, decode the glyph trace with the GlyphButton 1106 is released. The Mode Button 1108 can toggle between the twodifferent modes. For example, the two different modes can include thelinear mode and the pointing mode. In the linear (one to one) mode, theuser hand motion can be translated into cursor movement with highfidelity. In the pointing mode, the user's hand jitter is stronglyrejected so that it is easy for users to point to small icons, even ifthe user's hand has a strong jitter.

For example, the two different operating modes (e.g., the linear modeand the pointing mode) are suitable for different application. Forexample, the pointing mode can be utilized in applications that need tohandle small icons. Additionally, for example, the linear mode can beused in drawing and/or glyph applications that do not need to handlesmall icons.

Although illustrated in FIG. 11 is a remote control that is associatedwith four buttons, a remote control utilizing the state machine can haveany number of buttons. For example, the remote control can have threebuttons—e.g., gesture, glyph and select—with an automatic context-awaremode switch algorithm. In another example, the remote control can havetwo buttons—e.g., gesture/glyph and select—where the gesture recognitioncan also recognize glyph. The remote control can also be a one buttonremote control, where a quick button press can be treated as select, anda longer button press (e.g., press and hold) can be treated asgesture/glyph.

In the example of the one button remote control, the button on thedevice can have two purposes: when the electronic device is off, abutton press will turn the electronic device on. When electronic deviceis on, pressing and holding the button will allow the user to entergesture commands, and the remote control can capture the user's handmovement (e.g., via three axis motion sensors, including accelerometer,gyroscope and/or compass). The captured hand movement can be used forgesture recognition (e.g., by a processing unit inside the remotecontrol device).

A set of gesture commands can be assigned to represent control functionsfor the electronic device. For example, in the case of a television asthe electronic device, a circle gesture can bring up the TV menu, an “x”gesture can turn the TV off, a rolling action can turn the volume up ordown, a “>>” gesture can make the TV fast forward, a “∥” gesture canpause the play, and so on. The gesture command definition can changebased on the display mode. For example when the TV menu is displayed onthe screen, the up and down pitch movement can scroll the menu. If theTV is playing a movie, the same up and down pitch movement can beinterpreted as the “∥” gesture, which pauses the play. A user can alsoassign a set of handwritten numbers and/or letters as shortcuts. Forexample, a user can assign the writing of a “C” as an instruction tochange the channel to CNN.

The captured hand movement can also be interpreted as a cursor movement,in other words, the pointing mode. The cursor movement can be used toinput search text on am on-screen keyboard, or select a menu item, forexample. The interpretation of the captured hand movement (either the“gesture” mode or the “pointing” mode) can be based on the display mode,or based on an explicitly defined “mode switch” gesture. A “mode switch”gesture can be a double yaw shake, a double tap, or any other gesturethat is easily recognized, but hard to be interpreted as another gesturein the existing command set.

The button on the remote control can be replaced by a capacitive sensorand/or a pressure sensor (e.g., the remote control can include nobuttons on the surface, and the surface can be soft and able to besqueezed so that when the remote control is squeezed, the gesturecommand button is considered pressed). This can allow the remote controldevice to distinguish not only whether the button is pressed, but alsohow hard the button is pressed. The extra resolution of button pressstrength can be used, for example, to control the sensitivity of themotion sensors, the moving speed of the cursor, the connecting movementbetween multiple strokes, and the like. The remote control can alsoinclude a microphone, so that the user can input a voice command.

Referring now to FIG. 12, illustrated is a system block diagram of astate machine 1200. FIG. 12 shows an exemplary embodiment of a mousecursor display state machine. According to an embodiment, the statemachine 1200 can have three modes that can indicate a state of a mousecursor. The modes can include pointing 1202, moving 1204, and drawing1206.

During the pointing 1202 state, a cursor is locked at a certainposition. In the pointing 1202 state, hand jitter and other artifactscan be eliminated from the motion. The motion can be detected and thehand jitter avoided by a filtering technique. A center line can becalculated through a low pass filter. A moving window can be calculatedbased on the center line. A peak value can be captured during the movingwindow to serve as an adaptive threshold. If the grid is moved outsidethe threshold, the state can change from the pointing 1202 state to themoving 1204 state.

The moving 1204 state is a free running linear (one to one) mode inwhich the cursor movement is directly based on a hand motion. If abutton is pushed, for example for a certain time (e.g., 0.2 seconds),the state can transition to the drawing 1206 state. In the drawing 1206state, the grid trajectory can be linearly mapped to the cursor display.The button release can be predicted and the distortion by the buttonrelease can be avoided.

In addition to the state of the mouse cursor, the state machine can alsoinclude two system modes. Referring now to FIG. 13, illustrated is asystem block diagram of an embodiment of a state machine 1300 includingthe system modes.

For example, the system modes can include a gesture mode 1302 and aglyph mode 1304. In the gesture mode 1302, the cursor can be frozen anddifferent gesture commands control different aspects of the system. Forexample, in the gesture mode 1302, three different gesture commands caneach be associated with a distinct gesture.

In the glyph mode 1304, the cursor can move along with the remotecontroller device. The trajectory can be recorded in memory and decoded.For example, the glyph mode 1304 can apply to a hand written character.When the gesture indicating a handwritten character is saved in memory,the handwritten character can be decoded.

Referring now to FIG. 14, illustrated is an embodiment of a statemachine 1400. For example, the state machine 1400 can be a pointeralgorithm state machine. The algorithm includes a program (e.g., logic)that is executable by a processor. A benefit of a state machine is todecompose complex logic into a set of manageable sub-logic.

The state machine 1400 can be decompressed into three sections: a frozencursor section 1402, a moving cursor section 1404, and a transactionalsection 1406. Upon initial start up, the state machine can initialize inan initial state (not shown) that can detect either the cursor moving orthe cursor not moving. The initial state can be a state for bothstarting and finishing a state for the pointer algorithm. For example,the initial power-on can make the state machine run from the initialstate. Any sequence of the pointer algorithm can finish in the initialstate.

The frozen cursor section 1402 is a state where the cursor is notmoving. The frozen cursor section 1402 can include a pointing state1408, a button click handling state 1410, and a gesture state 1412. Inthe frozen cursor section 1402, the cursor is frozen in place and anydetected hand jitter is rejected and/or eliminated.

The moving cursor section 1404 is a state where the cursor is moving.The moving cursor section 1404 can include a cursor moving state 1414, adrawing state 1416 and a glyph state 1418. In the moving cursor section1404, the cursor directly follows the hand motion, so hand jitter neednot be eliminated. The transactional state 1406 can manage a transitionfrom the moving section 1404 to the frozen cursor section 1402.

Referring now to FIG. 15, illustrated is a simplified state transitiondiagram 1500 that can be implemented by an embodiment of a statemachine. Elements 1504, 1508, 1516 and 1520 are highlighted illustratingthat the cursor is frozen in the highlighted states so that user handjitter and other artifacts can be rejected. In the other(non-highlighted) states, the cursor can follow the user hand movement.An advantage of the state transition architecture is a decomposition ofcomplex logic into a set of manageable sub-logic. FIGS. 16-19 illustratestate transition diagrams 1600-1900 for sub-logic related to the complexlogic illustrated in FIG. 15.

Referring to FIG. 16, illustrated is a state transition diagram 1600 forthe CURSOR_INIT 1502 state. The pointer algorithm both starts andfinishes with the CURSOR_INIT 1502 state. For example, the state machinecan run from the CURSOR_INIT 1502 at an initial power on. Additionally,for example, CURSOR_INIT 1502 is the end state for any other sequence.There are two branches from CURSOR_INIT 1502: CURSOR_POINTING 1502 andCURSOR_MOVING 1506.

The state machine can enter the CURSOR_MOVING 1506 state when the handmotion is detected. The state machine can enter the CURSOR_POINTING 1504state when hand motion is not detected.

In the CURSOR_POINTING 1504 state, a user intention has been determinedas pointing to a target icon. The cursor is frozen, which allows for theelimination of hand jitter. Accordingly, any hand motion subsequent isregarded as jitter as long as there is no constant direction for thehand motion. Accordingly, if hand motion in a constant direction isdetected, the state machine transitions to the CURSOR_MOVING 1506 state.

Referring now to FIG. 17, illustrated is a state transition diagram 1700for the CURSOR_POINTING_BUTTON 1508 state. The state machine is drivento enter the CURSOR_POINTING_BUTTON 1508 state when a selection buttonis pressed. If the selection button is released, the state cantransition to the CURSOR_POINTING_DOUBLE_CLICK 1510 state to be ready toreceive another selection button click. If the selection button is notreleased within a certain time window, the user intent is detected asdrawing. The state is transitioned to CURSOR_DRAWING 1512.

With regard to CURSOR_POINTING_DOUBLE_CLICK, if the time threshold ispassed and no second button press has occurred, the click is complete,and the state is transitioned to CURSOR_INIT 1502. If a second buttonclick occurs before the time threshold, the double click processcontinues. They state is transitioned toCURSOR_POINTING_DOUBLE_CLICK_WAIT 1520 to wait for the button release.The presence of the second click can drive the state machine to theCURSOR_POINTING_DOUBLE_CLICK_WAIT 1520 state. The release of the buttonis expected in this state to complete the double click sequence. Afterthe button is released, the state is transitioned to CURSOR_INIT 1502.

Referring now to FIG. 18, illustrated is a state transition diagram 1800for the CURSOR_MOVING 1506 state. When a user intention is determined asmoving the cursor in a consistent direction, the state machine can bedriven to the CURSOR_MOVING 1506 state. In the CURSOR_MOVING 1506 mode,the cursor can track the hand movement on a one to one basis.

If no motion is detected, the state can be transitioned toCURSOR_TRANSITION_MOVING 1514, which is ready to freeze the cursor. If adrawing action, a gesture action or a glyph action is detected, forexample through a corresponding button, the state can be transitioned toa respective state 1512, 1516, or 1518.

The state machine can enter the CURSOR_DRAWING 1512 state upon detectionof a drawing action (e.g., a drawing button press). The CURSOR_DRAWING1512 state handles the drawing function, for example, with regard to apainting program, a Chinese hand writing software or the like. After abutton release is detected, the drawing process is completed. The statecan transition to CURSOR_INIT 1502.

The state machine can enter the CURSOR_GESTURE 1516 state upon detectionof a gesture action (e.g., a gesture button press). After a buttonrelease is detected, the gesture process can be completed and the statecan be transitioned to CURSOR_INIT 1502.

The state machine can enter the CURSOR_GLYPH 1518 state upon detectionof a glyph action (e.g., a glyph button press). After a button releaseis detected, the glyph process can be completed and the state can betransitioned to CURSOR_INIT 1502.

Referring now to FIG. 19, illustrated is a state transition diagram 1900for the CURSOR_TRANSITION_MOVING 1514 state. TheCURSOR_TRANSITION_MOVING 1514 state is a transition state from cursormoving to cursor pointing. When a user intent is detected as pointing toa target icon, the state machine needs additional time to conform thisdetection. After the confirmation and/or approval of the user intent,the state machine can be transitioned to the CURSOR_POINTING 1502 state.

Referring now to FIG. 20, illustrated is a schematic system blockdiagram of an example motion processing function 2000 for a remotecontrolling device. At element 2002, a user can make a hand motion,which is detected by one or more motion sensors at element 2004. Datafrom the sensors is sent to a processing unit where rotational movementis calculated and transformed into three cursor location coordinates2006, while hand jitter is removed 2008. A state machine 2010 cantransition into different states, for example, based on an input fromone of several buttons 2012 on the remote controlling device. When theuser pushes the buttons, some unwanted hand motion will be induced,which will disturb the cursor on the screen. In order to remove theunwanted motions, an intelligent state machine is utilized that canengage in intelligent motion tracking 2014 and feedback 2016.

While the various embodiments have been described in connection with thevarious figures, it is to be understood that other similar embodimentsmay be used or modifications and additions may be made to the describedembodiment for performing the same function without deviating therefrom.Therefore, the present innovation should not be limited to any singleembodiment, but rather should be construed in breadth and scope inaccordance with the appended claims.

What is claimed is:
 1. A handheld remote control device, comprising: atleast one three axis gyroscope that senses a first three-dimensionalmovement of the handheld remote control device and provides a firstthree-dimensional data output; at least one three axis accelerometerthat senses a second three-dimensional movement of the handheld remotecontrol device and provides a second three-dimensional data output; anda processing unit that uses the first and second three-dimensional dataoutputs to generate information associated with one of cursordisplacement for a cursor and cursor location for the cursor; wherein,the at least one three axis accelerometer, the at least one three axisgyroscope and the processing unit are integrated in a single integratedcircuit chip.
 2. The handheld remote control device of claim 1, whereinthe processing unit converts, to two-dimensional data, the first andsecond three-dimensional data outputs, wherein the first and secondthree-dimensional data outputs correspond to a hand motion of a hand ofa user that is holding the handheld remote control device.
 3. Thehandheld remote control device of claim 1, wherein the firstthree-dimensional movement comprises a rotational movement and istransformed into cursor location coordinates.
 4. The handheld remotecontrol device of claim 3, wherein the rotational movement is linearlymapped to a two dimensional cursor location represented by a yaw and apitch.
 5. The handheld remote control device of claim 1, wherein a setof gesture commands represents control functions for an electronicdevice.
 6. The handheld remote control device of claim 5, wherein theelectronic device comprises a television (TV), and wherein the controlfunctions comprise any of: bring up a TV menu, turn the TV off, turn avolume of the TV up; turn the volume of the TV down, make the TV fastforward, pause the TV, scroll the TV menu.
 7. The handheld remotecontrol device of claim 1, wherein the handheld remote control deviceoperates in at least two modes of functionality, wherein a first mode ofthe two modes of functionality includes a tracking mode wherein theinformation associated with the cursor location directly tracks thefirst and second three-dimensional data outputs, and a second mode ofthe two modes functionality includes a non-linear mode wherein aprocessor generates the information associated with the cursor locationby removing unwanted hand motion of a user.
 8. The handheld remotecontrol device of claim 7, further comprising: an intelligence componentthat switches between the at least two modes of functionality.
 9. Thehandheld remote control device of claim 7, further comprising: theprocessor that switches between the at least two modes of functionalityin response to occurrence of an input condition.
 10. The handheld remotecontrol device of claim 9, wherein the occurrence of the input conditionincludes one of the user: making a gesture while holding the handheldremote control device; tapping the handheld remote control device,shaking the handheld remote control device; and drawing a handwritingsymbol while holding the handheld remote control device.
 11. Thehandheld remote control device of claim 9, further comprising: theprocessor that changes a shape of the cursor in response to anoccurrence of the input condition.
 12. The handheld remote controldevice of claim 9, further comprising: the processor that initiates anoperational control of the handheld remote control device in response toan occurrence of the input condition.
 13. The handheld remote controldevice of claim 9, wherein switching between the two modes offunctionality comprises a gesture based mode switching; and wherein thegesture comprises any of: vertically flipping the handheld remotecontrol device; pointing the handheld remote control device up to thesky; pointing the handheld remote control device down to the ground; andshaking the handheld remote control device a certain number of times.14. The handheld remote control device of claim 9, wherein the handheldremote control device comprises a gesture command button; wherein thegesture command button is operable to switch between the two modes offunctionality; and wherein the gesture command button further comprisesany of a capacitive sensor and a pressure sensor.
 15. The handheldremote control device of claim 14, wherein the handheld remote controldevice comprises a squeezable surface; the squeezable surface isoperable to switch between the two modes of functionality; wherein thesqueezable surface on the handheld remote control device comprises anyof a capacitive sensor and a pressure sensor; and the gesture commandbutton is considered pressed when the handheld remote control device issqueezed.
 16. The handheld remote control device of claim 7, wherein anunwanted hand motion is removed by using an intelligent state machine,and wherein the unwanted hand motion comprises any of hand jitter andbutton push motion.
 17. The handheld remote control device of claim 7,wherein a presence of the unwanted hand motion results in freezing thecursor.
 18. The handheld remote control device of claim 17, wherein thepresence of the unwanted hand motion is determined by comparingdisplacement of a sensed movement from a target to a boundary of apredetermined window, in a predetermined number of samples in onedirection.